Fix premature recycling of GDS staging buffers - #6624

Category:

Bug fix (non-breaking change which fixes an issue)

Description:

Fix premature reuse of GDS staging buffers that can corrupt GPU NumPy reader output.

wait_buffers() releases the mutex while synchronizing the reusable CUDA event. Another thread can reclaim the completed batch, submit a newer batch, and re-record the event before the waiter reacquires the mutex. Unconditionally reclaiming the current scheduled_ list then exposes buffers still used by GPU copies.

After reacquiring the mutex, use return_ready() to check the current event recording before reclaiming its buffers. The existing acquisition loop waits again if necessary.

Fixes #6623.

Additional information:

Affected modules and functionalities:

GDS staging-buffer reclamation used by readers.numpy(device="gpu").

Key points relevant for the review:

The wait remains outside the mutex. Event querying and reclamation occur under the mutex, keeping the event recording consistent with the scheduled-buffer list.

Tests:

• Existing tests apply
  • GDSMem.StagingEngine and GDSMem.StagingEngineBigTest (not run locally).
• New tests added
  • Python tests
  • GTests
  • Benchmark
  • Other
• N/A

New test: GDSMemTest.ConcurrentStagingBufferReuse runs eight readers against a two-buffer pool with immediate copy submission and checks every output byte. It uses only the engine's public API, with no production hooks or timing sleeps. Detection depends on thread scheduling.

Validation:

• Focused C++ lint and git diff --check passed. The complete modified gds_mem_test.cu translation unit compiled with NVCC.
• Built that test translation unit against official DALI 2.1.0 and ran the new test outside the sandbox: 10/10 runs detected output corruption with the unmodified library; 10/10 passed with the exact patched wait_buffers() method interposed. The unmodified case used no method interposition or scheduling hooks.
• Repeated the extracted test body separately: 10/10 failures without the fix, 10/10 passes with the fix.
• Ran the controlled reproducer from [GDS] wait_buffers can prematurely recycle in-flight buffers and corrupt NumPy reader output #6623 against official DALI 2.1.0: original implementation exposed 32 pending buffers and corrupted one complete 2 MiB output block; the event-recheck variant exposed zero pending buffers and had zero corrupt values. Neither watchdog fired.
• Repeated the fixed case using the exact wait_buffers() method extracted from this branch: zero premature buffers and zero corrupt values.
• Runtime validation uses method interposition with real CUDA copies and cuFileRead, with CUFILE_FORCE_COMPAT_MODE=true. A full DALI rebuild and end-to-end NumPy pipeline validation were not performed.

Checklist

Documentation

• Existing documentation applies
• Documentation updated
  • Docstring
  • Doxygen
  • RST
  • Jupyter
  • Other
• N/A

DALI team only

Requirements

• Implements new requirements
• Affects existing requirements
• N/A

REQ IDs: N/A

JIRA TASK: N/A